Phrase-Level Temporal Relationship Mining for Temporal Sentence Localization

In this paper, we address the problem of video temporal sentence localization, which aims to localize a target moment from videos according given language query. We observe that existing models suffer sheer performance drop when dealing with simple phrases contained in sentence. It reveals limitation only capture annotation bias datasets but lack sufficient understanding semantic To problem, propose phrase-level Temporal Relationship Mining (TRM) framework employing relationship relevant phrase and whole have better each entity Specifically, use predictions refine sentence-level prediction, Multiple Instance Learning improve quality predictions. also exploit consistency exclusiveness constraints regularize training process, thus alleviating ambiguity prediction. The proposed approach sheds light on how machines can understand detailed their compositions generality rather than learning biases. Experiments ActivityNet Captions Charades-STA show effectiveness our method both localization enable model interpretability generalization unseen seen concepts. Code be found at https://github.com/minghangz/TRM.